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ASYMPTOTIC BOUNDARY FORMS FOR TIGHT GABOR FRAMES 
AND LATTICE LOCALIZATION DOMAINS 

H.G. FEICHTINGER, K. NOWAK, AND M. PAP 


Abstract. We consider Gabor localization operators G^^q, defined by two param¬ 
eters, the generating function ^ of a tight Gabor frame {(/'aIagA) parametrized by 
the elements of a given lattice A C , i.e. a discrete cocompact subgroup of , 
and a lattice localization domain D C R.^ with its boundary consisting of line seg¬ 
ments connecting points of A. We find an explicit formula for the boundary form 
BF{4>, D) = Aa lim/j^oo PP{G^,Rn) ^ normalized limit of the projection functional 
PF{G^^q) = ~ ^i(C 0 .n)), where Xi{G^fl) are the eigenvalues of 

the localization operators G^^q applied to dilated domains Rfl, R is an integer and 
Aa is the area of the fundamental domain of the lattice A. Although the lattice A 
is also a parameter of the localization operator we assume that it is fixed and we do 
not list it explicitly in our notation. The boundary form expresses quantitatively the 
asymptotic interactions between the generating function of a tight Gabor frame 
and the oriented boundary dQ of a lattice localization domain from the point of 
view of the projection functional, which measures to what degree a given trace class 
operator fails to be an orthogonal projection. It provides an evaluation framework 
for finding the best asymptotic matching between pairs consisting of generating func¬ 
tions (j) and lattice domains D. It takes into account directional information involving 
outer normal vectors of the linear segments constituting the boundary of D and the 
weighted sums over the corresponding half spaces of the absolute value squares of 
the reproducing kernels obtained out of ^. In the context of tight Gabor frames and 
Gabor localization domains placing an upper bound on the square of the norm of 
the generating function (j) divided by the area of the fundamental domain of lattice 
A corresponds to keeping the relative redundancy of the frame bounded. Keeping 
the area of the localization domain D bounded above corresponds to controlling the 
relative dimensionality of the localization problem. 


1. Main Results and Their Context 

We start by formulating the main results of this paper. Let A C be a lattice, i.e. 
a discrete cocompact subgroup of , satisfying condition Aa < 1, where Aa denotes 
the area of the fundamental domain of A. For a function 0 G L^(R) and A = (g,p) 
we dehne — q). Let the system {^aIasA’ where 0 G L^(R) is a hxed 


1991 Mathematics Subject Classification. 47B35,42G15,81Q20. 

Key words and phrases. Toeplitz operators, phase space localization, tight Gabor frames, semi- 
classical limit. 


1 




2 


Asymptotic Boundary Forms 


function, be a tight Gabor frame, i.e. we assume that for every / G L^(R) 

/ = X](/,0a)0a, (1) 

AeA 

where the convergence of the sum is understood in the unconditional norm sense. 
Function 0 is called the generating funetion of a tight Gabor frame {^AlyeA- 
operator acting on and of the form 

G«n = 5]ii(A)(/,<#'A)^A (2) 

aga 

is called a Gabor multiplier of localization type, if its symbol b, defined on the lattice 
A, is non-negative and summable. It can be easily verihed that Gabor multipliers of 
localization type are non-negative, trace class, and that 


aga 


and the operator norm of satishes 

< ||6||;oo(a). 

The projection functional PF is dehned on positive dehnite, trace class operators T, 
with their operator norm bounded above by 1, via the formula 


PF(T) = Y,HT){1-HT)), (3) 

i=0 

where {Ai(T)}“Q are the eigenvalues of T. The projection functional measures the 
extend by which the operator T fails to be an orthogonal projection. It takes non¬ 
negative values and it vanishes on the space consisting of hnite dimensional orthogonal 
projections. A collection of line segments h C M^, f = 1, 2,..., C each of them starting 
and ending at a point of A is called a A cycle if the union /i U /2 U ... U /c forms 
a closed continuous line without self intersections. A bounded, connected and closed 
subset G C is called a A domain if its boundary consists of a hnite family of A 
cycles Ci, i = 1,2,..., B satisfying condition dist(Gi, Cj) > 0 for i ^ j. We say that a 
function 0 G L^(M) satishes condition <h if 


$: ^ |(0,0A)nA| < oo. 

aga 


For a A domain G C and a generating function 0 of a tight Gabor frame {0a}aga 
satisfying condition <F the boundary form BF(0, G) is dehned by the formula 

N 

BF{(t),Q) = E length(/0 ^ ^ dist(A, 1 (0, 0a) I , (4) 

i=i xeUi 

where U, i = are the line segments constituting the boundary of G, Ui is 

the unit vector orthogonal to directed outside G, Pi = {w G | tc • Uj = 0}, 
Ui = {A G A|A'?rj ^ 0]-. 
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The projective metaplectic representation defined on S'L(2,M) provides the most 
natural way to deal with linear changes of coordinates of the time-frequency plane 
Its definition and its basic properties are reviewed in Section 4. It came to 
us as a surprise, that the boundary form BF, representing the limit value of the 
projection functional PF with respect to dilation factors of the localization domain 
tending to infinity, is invariant with respect to the action of the projective metaplectic 
representation. Many geometric and numerical studies of lattices in distinguish 
the hexagonal lattice as being special, but this is not the case in the context of the 
projection functional PF and its limit value BF. 

In our first result we describe the invariance properties of PF and BF with re¬ 
spect to the projective metaplectic representation. Although the invariance of PF is 
standard we include it as well for the sake of completeness. 

Theorem 1.1. Let /i he the projective metaplectic representation of SL{2,M.) acting 
on L?{Mj).Let cf be a generating function of a tight Gabor frame {^aIasa satisfying 
condition <h with respect to A. Then for any A G S'L(2,M) T = A(A) is a lattice, 
{{pi{A)(l))^}^^Y is a tight Gabor frame with the generating function ti{A)(j) satisfying 
condition $ with respect to T, and 

(i) for any Gabor multiplier of localization type with the symbol bounded above by 
1 7 G^(A) 4 >,hoA-^ is a Gabor multiplier of localization type with the symbol bounded above 
by 1, and 

PF {G(f,,b) = PF [Gf^(A)(f>,boA-^) , (5) 

(a) for any fl a A domain, A(12) is a T domain, and 

BF{(t), n) = BF{ia{A)(j), A(12)). (6) 

Our second result is the principal result of the current paper. It describes the limit 
behavior of the projection functional PF applied to a Gabor multiplier with the symbol 
of the form bji = xnn, i-e. the characteristic function of a dilated lattice domain 12 
by a factor i? G Z, with i? —)■ oo. Gabor multipliers of this special form are called 
Gabor localization operators and they are denoted as G^^rq. We will see later on that 
the dilation factor R can in fact take any real values. 

Theorem 1.2. Let (p be a generating function of a tight Gabor frame parametrized by 
lattice A and satisfying condition <h, and 12 a A domain contained in . Then 

PF{Gt.m) ( 7 ) 

R-^oo Jri Aa 

Formula ([7]) expresses the limit behavior of the projection functional PF in terms of 
the boundary form BF. It provides a very explicit, quantitative way of describing the 
interactions between the boundary dVt and the reproducing kernel obtained out of the 
generating function (p. 

The invariance of the projection functional PF and the boundary form BF with 
respect to the action of the projective metaplectic representation expressed in Theorem 
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[EH taken together with the limit result of Theorem ll.21 has its important consequences. 
We can conclude that no lattice A is distinguished, neither from the point of view the 
value of the limit of as i? —)■ oo, nor from the point of view of its rate of 

convergence. 

Corollary 1.3. For any lattice A C satisfying condition Aa < 1, any generat¬ 
ing function (f) of a tight Gabor frame {^yjAeA; A lattice domain 12, and any 
a,h > 0 satisfying ab = Aa, there are a generating function (f of a tight Gabor 
frame {^\}x£ai,xbz and a aZ x bl. lattice domain 12, satisfying ||0||l2(r) = ||0||i,2(r); 
Area(12) = Area(n), PF^G^jj^Ro) = PF{G^ rq) , for all R > 0, and also BF^cf^Vt) = 

BF{(j),Cl). The rates of convergence of -PBF{(j),Q,), and —to 

■PBF^f), n) are the same. 

Many important mathematical theories started in the one dimensional setup, where 
a multitude of additional tools is available, and then through various stages of evo¬ 
lution came up into their full form in any hnite dimension. This was the case of the 
representation theory of semi-simple Lie groups, which started with the listing of all 
irreducible representations of SL{2, M). Classical time-frequency localization operators 
allow explicit diagonalization in one dimension, but not in higher dimensions, yet it 
was possible to transfer a large portion of one dimensional results to higher dimen¬ 
sions. In one dimension, for several classes of potentials, Schrodinger operators can 
be treated via explicit formulae, yet many results that follow the guidelines of the one 
dimensional setup are also true in higher dimensions, although explicit formulae are 
no longer available. The development of the theory of Gabor frames follows a simi¬ 
lar path. As this is the case of many relatively recent theories, some of its branches 
are still at an early, one dimensional stage, e.g. the treatment of Gabor frames with 
maximal lattice parameter frame set via totally positive functions of hnite type done 
recently by Grochenig and Stockier in [23]. In many instances the distinction between 
one and higher dimensions is related to the differences between the theories of one and 
several complex variables. The development of the phase space theory of reproducing 
formulae shows many similarities with the development of the theory of Gabor frames. 
The semidirect product xi Sp{n, M), where Sp{n, R) is the symplectic group con¬ 
sisting of 2n X 2n invertible matrices preserving the symplectic form, together with the 
extended projective metaplectic representation dehned on it, constitute the group of 
affine transformations of the phase space, acting geometrically on , and analytically 
on . The affine transformations of the phase space provide a natural framework 

for the constructions of reproducing formulae. All reproducing formulae coming out of 
connected Lie subgroups of x Sp{n, M) were characterized by De Mari and Nowak 
for n = 1 in [8], but no analogous results are known in higher dimensions. In dimen¬ 
sion one S'p(l,M) = S'L(2,M) and it is possible to obtain the list of all connected Lie 
subgroups of x S'L(2,M) out of the classical structure results describing the orbits 
of inner automorphisms of S'L(2,M). No analogous lists of subgroups are available in 
higher dimensions. In the current paper we deal with tight Gabor frames parametrized 
by a lattice A C Any tight Gabor frame parameterized by a lattice A C can 
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be transferred into a tight Gabor frame parameterized by a separable lattice aZ x 
a, 6 > 0, by an appropriate SL{2, M) linear transformation of the time-freqnency plane. 
We treat the case of a separable lattice via an explicit compntation and then we transfer 
the formnla we obtain to an arbitrary lattice with the help of the projective metaplectic 
representation. This approach does not generalize to higher dimensions. 

The principal resnlts of the cnrrent paper deal with discrete one-dimensional setnp 
of tight Gabor frames, therefore for the sake of consistency we formnlate dehnitions 
and reference resnlts only in one dimension. Operators of composition of convolntion 
with g followed by a pointwise mnltiplication by /, where both fnnctions / and g are 
dehned on the real line M, have integral kernels of the form 

Kix,y) = f{x)g{x-y). (8) 

They are commonly called convolntion-prodnct operators. Historically three dimen¬ 
sional convolntion-prodnct operators played an important role in the stndy of Schrodinger 
operators. The Birman-Schwinger principle allows a transition from a Schrodinger op¬ 
erator to a convolntion-prodnct operator. Ont of that transition it was possible to ob¬ 
tain sharp estimates for the nnmber of bonnd states of the Schrodinger operator. Glas- 
sical time-freqnency localization operators are one dimensional convolntion-prodnct 
operators with / and g characteristic fnnctions of intervals, where g is the inverse 
Fonrier transform of g. Their spectral properties were carefnlly stndied many years 
ago by Landan, Poliak, Slepian and Widom (see 0 . ra. I 2 H 1 , and the references pro¬ 
vided within), yet till now these classical results bring important ingredients for both 
theoretical and applied components occurring in many recent developments. There 
were also extensive studies of convolution-product operators with one of the function 
parameters f or g hxed and of prescribed potential type (see the book by Mazya [53]). 
but the dependence on both parameters, and the mutual interaction between / and g 
in the general case seems to be a difficult problem that is to a large degree still open. 

We are interested in operators with integral kernels of the form (| 8 ]), where transla¬ 
tions constituting the convolution with g are extended to combined actions of transla¬ 
tions and modulations of the Schrodinger representation, and applied to a function cj). 
As a consequence of this extension we need to introduce two arguments of the multi¬ 
plier /, one corresponding to translations, the other to modulations, and the extended 
integral kernel (|H]) becomes 

K{{q,p),y) = f{q,p)(j)q,p{y), (9) 

where (t>q,p{x) = — q) . We add complex conjugate over because we 

want our operators to be exactly square roots of Gabor-Toeplitz operators. The Gabor 
reproducing formula has the form 

/ = / {fAq,p)4>q,pdqdp, ( 10 ) 

where f,4> & L^(M), || 0 ||£, 2 (r) = 1 , and the convergence of the integral is understood 
in the weak sense. Introducing into flTOll a weight function b{q,p), called a symbol we 
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obtain a Gabor-Toeplitz operator 


T^,bf 


4>q,p)(l^q,p dqdp. 


( 11 ) 


Gabor-Toeplitz operators generalize Fock space Toeplitz operators. The Bargmann 
transform provides their mutual unitary equivalence, in case the normalized Gaussian 
is chosen for the generating function. Books by Folland [19] and Zhu [40] are very 
good references on the subject. In the held of phase space analysis Gabor-Toeplitz 
operators were introduced by Ingrid Daubechies. Her book [H] provides a comprehensive 
account of the background and the initial results. It is possible to study Gabor-Toeplitz 
operators in a very general context, with various classes of symbols and acting on a 
wide range of function spaces. It is convenient to discuss Gabor multipliers and Gabor- 
Toeplitz operators in parallel. In this paper we restrict attention to Gabor-Toeplitz 
operators of localization type acting on L^(M), i.e. we assume that the symbol b is 
non-negative, bounded and integrable with respect to the Lebesgue measure on M^. 
It is straightforward to verify that under these assumptions is non-negative, trace 
class. 




b{q,p) dqdp, 


the operator norm of satishes 






and that if |/(q',p)P = b{q,p), then the composition of the operator dehned by the 
integral kernel (19|) with its conjugate equals T^^b dehned in ffTTj) . Kernels of the form 
(1^ represent square roots of Gabor-Toeplitz operators flTT|) . They provide a link be¬ 
tween Gabor-Toeplitz operators and generalized convolution-product operators, where 
translations are substituted by actions of unitary representations. 


Asymptotic properties, as i? —)■ oo, of the symbolic calculus of Gabor-Toeplitz op¬ 
erators where b is integrable, 0 < b{q,p) < 1, and bR{q,p) = is the L°° 

normalized dilation of b, were studied in PI- For h a continuous function dehned on 
the closed interval [0,1] the operator hiT^^bn) is dehned via the spectral decomposition 
of ^ Szego type formula for operators of the form hiT^^bn) was obtained in [13], 

showing that 

lim = f b{q,p)h{b{q,p)) dqdp. (12) 

R^oo 

We need to multiply hiT^^bn) by T^^bn in formula flT^ in order to ensure that the oper¬ 
ator is trace class. The Bohr correspondence principle was concluded as a consequence 
of (IT^ . It asserts that in the normalized limit with factor in the denominator both 
the distribution of the eigenvalues of 

N{5,R) = |{* I HT^,bn) > 5}|, 0 < 5 < 1, (13) 

and the size of their plunge region 

M(5i,52,i?) = |{^ I < A(T^,b^) < 52}| , 0 < < <52 < 1, 


(14) 



H.G. Feichtinger, K. Nowak, M. Pap 


7 


are expressed directly via the corresponding quantities of the symbol function, the 
distribution of b 

\{{q,p)\b{q,p) > 6}\, (15) 

and the Lebesgue measure of its plunge region 

\{{q,p)\Si <b{q,p) < 62 }]. (16) 

In the passage to the normalized limit it is necessary to assume that the level sets 
{{q,p) I b{q,p) = f}, t = 6 , 61,62 , have Lebesgue measure 0. We were able to deduce 
that asymptotically the best localization properties, i.e. asymptotically there are no 
eigenvalues in the open interval (0,1), occur for symbols being characteristic functions 
of measurable sets C . 

In the next step operators , with symbols b = xn, were studied directly without 
the asymptotic limit. Two-sided estimates of the size of the eigenvalue plunge region 

M{ 6 u 62 ,(j),n) = \{t\ 6 i < A,(T^,o) < ^2}! (17) 

expressed in terms of the area of a strip of hxed size R around the boundary dQ and 
uniform with respect to generating functions (j) and localization domains fl, of the 
form 

Cl 1(911)^1 < M{ 6 i, 62 ,(j),n) < C 2 |(afl)^|, (18) 

where (cm)-^ = {{q,p) \ dist((g,p), 917)| < R}, were obtained in |^. In order to get two- 
sided uniform positive constants ci,C2 it was necessary to assume uniform decay and 
non-degeneracy of the reproducing kernels obtained out of generating functions (j) and 
uniform access to localization domains 17 and their complements from the points 
near their boundaries dfl. The size R of the strip (917)^ around 917 is one of the 
uniform parameters controlling mutual interactions between the generating functions 
(j) and domains of localization 17. 

The next step in the study of mutual interactions between generating functions and 
domains of localization from the point of view of estimating the size of the eigen¬ 
value plunge region was accomplished in [M]. The eigenvalues of Gabor-Toeplitz lo¬ 
calization operators satisfy estimates 0 < Aj(T<^^oJ < 1. The projection 

functional PF{T^^^) = ~ '^i(7)/,,r2,)) provides an exact, quantitative 

way of measuring the size of the eigenvalue plunge region. It has the same form 
as in ([3]) in the discrete setup. We assume that 17 is a bounded domain with 
boundary. Symbol a denotes the arc length dehned on 917. Function n is the Gauss 
map, i.e. n{r,s) is the unit normal vector at (r, s) G 917 directed outside 17. By 
F,[, = {w G I tc • n = 0} we denote the linear subspace of consisting of vectors 
orthogonal to v, and Ft, = {tc G | tc • n > 0} is the half space inside with v 
being the inner normal vector. The boundary form is dehned as 

BF{(j),n)=[ [ \{(j),(j)g^p)\‘^dist{{q,p),Pn(r,s))dqdpda{r,s). (19) 

Jan 

Under an appropriate integrability condition imposed on 0, condition $ of the current 
paper is its adaptation to the discrete setup, that makes formula (IT^ well dehned it 
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has been shown in [3l] that 

lim ^ BF{ct),n). (20) 

R-^oo R 

Formula ([20]) expresses the limit behavior of the projection functional in terms of the 
boundary form. It describes quantitatively the interactions between the boundary dQ 
and the reproducing kernel obtained out of the generating function 0. Boundary form 
dl]) is an adaptation of (IT^ to the discrete setup. 

Let us £x the area A and a generating function 0 for which the boundary form fllOj) 
is well defined, and let us ask for what localization domains hi of area A the boundary 
form BF{(j), 12) takes the smallest possible value. Let us introduce surface tension 
defined on the unit sphere of the time-frequency plane , given by the formula 

= f |(0,0q,p)|Mist((g,p),P^)dgdp. (21) 

Function defined in fl?I|) . determines the Wulff shape 

= Pi {tc e I tc • n < M^{v)}. (22) 

For the given generating function 0 let us construct K^, dehned in fl22l) . and let us scale 
it with a constant c, so that the area of cK^ is A. It occurs, that, up to translation, 
the domain cK^ is the only domain among all domains 12 with hnite perimeter and 
area A for which BF{Q, 0) is minimal. The shape cK^ is the optimal domain of 
localization. 

The principal results of the current paper deal with the discrete setup of tight Gabor 
frames defined on L^(M). On the level of convolution-product operators with kernels 
of the form ([H]) the transition to the discrete setup translates to making the range 
variables discrete and keeping the domain variables continuous. The multiplication 
parameter / is now defined on the group of integers, but the convolution parameter g 
is still dehned on the real line M. After the modihcation the integral kernel (|8]) becomes 

K{m,y) = f{m)g{m-y). (23) 

In the discrete setup the Gabor reproducing formula fITU]) is substituted by a tight 
Gabor frame expansion ([T]) 

/ = X](/, 0a)0A, 

AeA 

and the Gabor-Toeplitz operator fITT]) by a Gabor multiplier ([2]) 

AgA 

Gabor multipliers were introduced as a phase space analysis tool parallel to Gabor 
expansions. Both Gabor-Toeplitz operators and Gabor multipliers are currently very 
actively investigated, mostly from the point of view of their usage in phase space anal¬ 
ysis. They were applied as phase space partitioning operators in [10] , leading to a 
new characterization of modulation spaces, as isomorphism maps, in [21], [2S], in the 
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context of modulation spaces and weighted Bargmann-Fock spaces, in 0. Eli, dl, 
0 . 01 as approximation blocks for the representation of Hilbert-Schmidt, pseudodif¬ 
ferential, and Fourier integral operators. A recent survey [15] provides an overview of 
the results on both Gabor-Toeplitz operators and Gabor multipliers from the point of 
view of phase space localization. Not all of the phenomena are the same for Gabor- 
Toplitz operators and Gabor multipliers. The cut-off phenomenon happens only in the 
continuous setup (see ra). the Berezin transform is invertible in a stable way only in 
the discrete setup (see HU)- 

We restrict attention to Gabor multipliers of localization type, i.e. we assume that 
the symbol b dehned on A is non-negative and summable. The operator given by the 
kernel 

K{\,y) = fWMii), 

a discrete analogue of (jH]), acting from L^(M) into /^(A), is again a square root of 
provided |/(A)|2 = 6(A). 

A discrete analogue of the Szego type formula flT^ applied to required us to 

assume that the symbol function b is dehned on that it is Riemann integrable, has 
compact support, and that 0 < b{q,p) < 1. Also in the discrete setup the Szego type 
formula captures essential asymptotic properties of the symbolic calculus of Gabor 
multipliers of localization type as i? —)■ cxd, where R is the parameter of the 
normalized dilation bR{q,p) = 6(^, ^). For any continuous function h dehned on [0,1] 
the operator h{G(j,fi^) is dehned in terms of the spectral decomposition of G^^bn- The 
discrete analogue of flT^ was obtained in [H]. It states that 

R^oo R^ Aa 

The motivation for both formulae fll2l) and fl2T|) came out of Widom’s paper [38] . The 
Bohr correspondence principle, i.e. the normalized limits, the same as for the eigenvalue 
distribution of (IT^ . and the size of their plunge region of (ITT]) , can be concluded for 
Gabor multipliers of localization type. In the discrete setup we need to insert the 

multiplicative constant —iii front of ffl^ and fflBl) . the same constant as in (1241) . 
but otherwise the normalized limits are same as in the case of Gabor-Toeplitz operators 
of localization type. In particular for a bounded, Riemann measurable set G we obtain 

,• |{^ I Ai(G</,,m) > 1 - 6}| ^ WMlhiM) 

R^oo R^ Aa 

for any e > 0, where in case bn = xrti we use the notation = G^^rq, . According 

to the principles of quantum mechanics we interpret the area |f2| as the dimension of 

the space of functions localized in phase space to G. The constant compensates 

the relative redundancy of the tight frame. 

Two-sided uniform estimates of the size of the eigenvalue plunge region, the discrete 
analog of flTTl) . were also obtained in ng. In the discrete context localization domains 
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f2 are finite subsets of A, the strips of size R around the boundary dVL occurring in 
ffTSD are substituted by 

= {A e I dist(A, < -R} U {A G I dist(A, Q) < R} , 
and |(Rf2)'^| by the number of points of A inside 

Out of Theorems 11.11 fL2] we were able to conclude Corollary 11.31 expressing the fact 
that from the point of view of the projection functional PF and the boundary form 
BF no lattice A is distinguished. It would be interesting to isolate those phase space 
phenomena that make distinction between lattices parameterizing tight Gabor frames. 
The book by Martinet [32] is a comprehensive resource on lattices in Euclidean spaces. 

We would like to indicate intuitive reasons explaining the existence of the normalized 
limit of the projection functional applied to . Let F be the absolute value square 

of the reproducing kernel, i.e. F{v) = |(0,The properties of the symbolic 
calculus of Gabor multipliers allow us to express the normalized projection functional 
as 

S (25) 

A 

1^1,1^2 € A 

We observe that Xmiiy) = Ao('^) ^md XRn<^{^) = Xn<^{^) y i-®- instead of stretching the 
domain R times and keeping the lattice hxed we may make the lattice R times denser 
and keep the domain hxed. What counts are the relative sizes. Expression ( 1251 ) may 
be rewritten as 

^ Xn4i^i)FiRii2i - i^2))Xn{T^2)- (26) 

Form 1 1261) of ([25]) is more convenient for a conceptual geometric interpretation. We 
see that for large values of R the main contribution comes from pairs z/i, 1 / 2 , G 
1^2 G G, which he close together. Therefore, we expect that the resulting limit will be 
interpretable as an appropriate boundary form. 

In the continuous case the Wulff shape fl22l) is the optimal localization domain. 
There is no analogue of it for lattice domains. We do not know how large is the class of 
localization domains for which the asymptotic boundary forms exist. We expect that in 
the general case the problem of existence of normalized limits of projection functionals 
has to be considered in parallel with the asymptotic properties of the counting function 
IA n RQ\ as i? —)■ 00 . Harmonic analysis background on the counting problem |A fl 
RQ\ as R ^ 00 , together with the original Hlawka result, and several developments 
that followed afterwards are presented in Stein’s book [37|. The paper by Nowak [SB] 
discusses more recent approaches to the topic. 

Books by Christensen [3|, Daubechies [6|, Flandrin [18], Grochenig [20], Wojtaszczyk 
[39] . and collected volumes by Feichtinger, Strohmer [T6] . m present a broad back¬ 
ground of phase space analysis techniques needed for the theory of Gabor expansions. 
Papers by Balan [1], Cassaza [2], Heil [26] illustrate several further aspects of Gabor 
analysis not treated directly in this paper. 
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2. Tight Gabor Frames, their Constructions and Properties 

Let "H be a Hilbert space. A family of functions {4>j}j^j C "H is called a frame of 
H if there are constants 0 < A, B < oo such that for any f E 

A\\f\\l<Y.W,^i)nf <Bml ( 27 ) 

For any frame of Ti it is possible to choose the largest constant A and the smallest 
constant B for which (1271) holds. These two extreme values of A and B are called the 
lower and the upper frame bounds of the frame If the lower and the upper 

frame bounds are equal then the frame is called tight. Any tight frame gives rise to a 
discrete reproducing formula. We may renormalize a tight frame {4>j}j^j and obtain 
the norm equality 

i&J 

which in turn via the polarization identity may be interpreted as a reproducing formula 

/ = (28) 

iGJ 

with the convergence of the sum understood in the unconditional norm sense. There 
is a canonical way of constructing a tight frame out of a frame. For any frame 
we dehne a frame operator via 

Sf = ^{fAj)H(f>j- (29) 

j&J 

Condition (1271) guarantees that the above sum is unconditionally convergent for any 
f eT-L and that the linear operator S dehned in fl2^ is positive dehnite, bounded and 
invertible on "H. It is straightforward to verify that the family is a tight 

frame on "H. 

Gabor frames have the form {^aIasA’ where 0 E T^(M), (pq,p{x) = <p{x — g), 
and A is a lattice in We assume that the system {0a}aga is a tight Gabor frame 
normalized in such a way that the reproducing formula (1281) holds. If necessary we 
multiply the generating function 0 by an appropriate normalization constant. The 
principles of quantum mechanics allow us to interpret Gabor frames as coverings of the 
phase space. Each frame element 0a corresponds to a phase space block of area 1. Let 
Aa denote the area of the fundamental domain of lattice A. If Aa > 1 then the lattice 
A is too sparse and it is not possible to cover the phase space with blocks of area 1. 
There are no Gabor frames if Aa > 1 . If Aa = 1 , this is the so called critical case, it is 
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possible to cover the phase space with blocks of area 1, but without any overlapping. It 
occurs that in this case a Gabor frame is automatically a Riesz basis, and a normalized 
tight Gabor frame is an orthonormal basis. A well known Balian-Low theorem applies 
to the critical case. It states that the generating function of a Gabor frame has strictly 
limited smoothness and localization properties. It is possible to interpret it as the 
outcome of the covering property with no overlapping, i.e. a partitioning of the phase 
space. From the point of view of modulation spaces, which require smoothness and 
localization, there are no Gabor frames in the critical case Aa = 1. If Aa < 1 then 
the lattice A is dense enough and it is possible to cover the phase space with blocks of 
area 1. There is also a sufficient overlap between the blocks that allows constructions 
of generating functions of Gabor frames with arbitrary smoothness and localization 
properties as measured by the scale of modulation spaces. The book by Grochenig [20] 
is a comprehensive resource on all these facts. Further on we assume that Aa < 1. 


Let Ws{q,p), s > 0, be a weight function dehned on by the formula 

W,iq,p)=(l + {\q\^ + \p\^y^y. (30) 

Let denote the space of Schwartz class functions and the space of tem¬ 
pered distributions. Let us select a non-zero function g G and dehne the the 

modulation space , 1 < p, g < oo, as the space consisting of those tempered 
distributions / G for which the norm 



\{f,9q,p)fWs{q,pydq] dp 


<ilv 


1/9 


(31) 


is hnite. If p = oo or g = cxd the integral norm is substituted by the essential supremum. 
The dehnition of the modulation space does not depend on the choice of function g. 
For different functions g the corresponding norms are equivalent. If p = g we write 
instead of 

Let us recall that a function 0 G L^(M) satishes condition <F with respect to lattice 
A if and only if < oo- Gondition <F is necessary and sufficient 

for the boundary form i?F(0, G) of formula (|1]) to be well dehned for any A lattice 
domain G. The next proposition formulates a sufficient condition on a function 0 G 
L^(M), expressed in terms of modulation spaces, for condition $ to hold. Our sufficient 
condition follows by a direct application of the Young’s inequality and the standard 
theory of modulation spaces presented in jJO]. We do not include its proof. 

Proposition 2.1. //0 G L^(M) then 0 satisfies condition <F. 


The hrst construction of tight Gabor frames in dimension 1 was obtained by Daubechies, 
Grossmann and Meyer in 1986. Tight Gabor frames were called painless nonorthogonal 
expansions back then. The construction produced generating functions with compact 
support either in position or in momentum and with arbitrary smoothness measured 
by the number of continuous derivatives, any or (7°° was possible. The book 
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by Daubechies |6] provides an excellent acconnt of the initial stages of the constrnc- 
tions of tight Gabor frames. Then came Wexler-Raz biorthogonality relations, Walnnt, 
Janssen representations, and Ron-Shen dnality principle dealing with Gabor frames 
and the frame operator in any hnite dimension. All these are very nicely presented in 
Grochenig’s book [20]. Tight Gabor frames and in particnlar canonical tight Gabor 
frames obtained via the action of where S is the frame operator, are of the 

principal interest from the point of view of this paper. Generating fnnctions being 
members of the modnlation space where JR is a snbexponential weight, are the 

bnilding blocks for all other modnlation spaces. Snbexponential weights are a natnral 
generalization of the standard weights Wg dehned in fl30D (see [22| for the dehnition, 
provided on page 10, and for explanations of their nsage in time-freqnency analy¬ 
sis). It is important to know how to constrnct generating fnnctions of tight Gabor 
frames, which are members of M^. The major resnlt in this direction was obtained 
by Grochenig and Leinert in [22], where they proved that, for any lattice A C 
the canonical tight frame operator is bonnded on M^. If we have a Gabor 

frame with the generating fnnction in M^, then we also have a tight Gabor frame 
with the generating fnnction in . The existence of a generating fnnction in of 
a Gabor frame, for any lattice A C with the volnme of the fnndamental domain 
satisfying VoR < 1, was established recently by Lnef [21]. Feichtinger and Kaib- 
linger examined, from the point of view modnlation spaces, continnity properties of 
the canonical dnal generating fnnction, i.e. g = Sg, where S is the frame opera¬ 
tor, with respect to the pertnrbations of the lattice. They proved in [T2] that the set 

^ I is a Gabor frame} is open and that the 

map (0, A) I—)■ 0 is continnons from i^^fo ^iv • This resnlt was recently extended 

to canonical tight generating fnnctions 0 = by Leinert and Lnef [3T] . 


3. Proofs of the Main Results 

We start by recalling the dehnitions of onr principal objects of interest. A tight 
Gabor frame 

{0A}AgA’ 

parametrized by a lattice A C , and normalized in snch a way that the reprodncing 
formnla ([T]) holds, is given. A Gabor mnltiplier of localization type (|2]) is constrncted 
ont of the reprodncing system {^aIasA’ ^ non-negative, snmmable, and bonnded 
above by 1 symbol b, dehned on A. It is dehned by the formnla 

aga 

The projection fnnctional (|3]) applied to has the form 

OO 

PF{G^,b) = - \{G^,b)), 

i=0 
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where are the eigenvalues of Condition $, the additional require¬ 
ment imposed on the generating function 0, XIaga < C) 0 , is a necessary 

and a sufficient condition for the boundary form (jl]) to be well defined for any A 
localization domain fl, 

N 

BF{(j),n) = length(/j) ^ ^ dist (A,-Pj) I (0, I , 

i=i \eUi 

where U, i = are the line segments constituting the boundary of Ui is 

the unit vector orthogonal to li directed outside ff, Pi = {w G | tc • rij = 0}, 
t/j = {A G A I A • rij > 0}. 

Now we are ready to present the proof of Theorem 11.11 

Theorem 1.1. Let /i be the projective metaplectic representation of SL{2,M.) acting 
on L^(M).Lef (f be a generating function of a tight Gabor frame {0a}a€A satisfying 
condition <h with respect to A. Then for any A G S'L(2,M) T = A(A) is a lattice, 
{(/x(A)0)g,}g,gr is a tight Gabor frame with the generating function pi{A)(l) satisfying 
condition <h with respect to T, and 

(i) for any Gabor multiplier of localization type with the symbol bounded above 
by 1 , is a Gabor multiplier of localization type with the symbol bounded 

above by 1, and 

PF {G<t>,b) = PF (G/i(A)(/),feoA-i) , 


(ii) for any Q a A domain, A(f2) is a T domain, and 


BF{(f>,n) = PF(/i(A)0, A(fl)). 


Proof. The transformation rule {ii) of Lemma 14.11 describing the effect of the con¬ 
jugation by the projective metalplectic representation, shows that (i), i.e. formula 


PF {G<f>,b) — PF (G;i(A)(/),feoA-i) , 

holds. 

In the remaining part of the proof we deal with {ii), i.e. with formula (jb]) 

BF{(j),n) = BF{ia{A)(j),A{n)). 

The group SL{2,'R) acts transitively on the collection of all lattices T C satisfying 
condition Ap = A\. Therefore, we may assume that A(A) = aZ x 6Z, where a, 6 > 0 
and ah = Aa- It is clear that for any A G SL{2,M.) A(12) is an A(A) domain. The 
proof of the BF invariance rule (|H]), describing the transition from A to aZ x 6Z via 
A, requires two ingredients, the transformation rule for the reproducing kernel 

(0,0a), (32) 

and for the geometric atoms 


flsA = length(s) ■ dist(A, 1 ), 


(33) 
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where A G A, s is a line segment with its endpoints being the points of lattice A, 
constituting a portion of the boundary of ff, I is the line parallel to s, and passing 
through (0,0). These two, fl3^ and (l33|) . constitute the basic building blocks of the 
boundary form. Lemma 14.11 (i) provides the formula describing the transformation of 
the reproducing kernel (l3^ under the action of the projective metaplectic representa¬ 
tion, it is enough to apply it. However, we need to deal with the geometric atoms fl33|) . 
via a direct computation. 

The linear map A G S'L(2,M) providing the transition from A to aZ x 6 Z satis- 
hes condition det(A) = 1, therefore it preserves the orientation of the boundary. It 
transforms a line segment /j of the boundary of H to the corresponding line segment 
A{li) of A(f2), and the half lattice Ui of A to the corresponding half lattice A{Ui) 
of A(A). The fact that A might not preserve orthogonality does not matter. Lemma 
14.11 (i) allows us to conclude that the transformation of the boundary form BF is 
correct as far as the reproducing kernel (l3^ is concerned. It also shows that condition 
4) transforms properly. We need to show that the geometric atoms of also 
transform properly, i.e. that 

tts^x = length(As) ■ dist( 7 , Al), (34) 

where 7 = AX. The Iwasawa decomposition of S'L(2,M) (see [30]) allows us to rep¬ 
resent A as PK, where P is upper triangular and K is a rotation matrix, both in 
S'L(2,R). Since K is unitary it preserves distances and angles. It is clear that the 
geometric atoms properly transform under K. It is therefore enough to consider the 
upper triangular P. In order to show that (I5T|) holds, we consider i?, the inverse of 
F, providing a transition from aZ x 6 Z to A. We verify that 


leiagth{Rsan,bm) ■ dist(F(afc, bl), Rlan,bm) = ab\nl - km\, (35) 


where R 


d e 
0 


, d > 0, e G M, San,bm IS the line segment connecting points 


(0, 0) and {an, bm ), lan,bm is the line containing San,bm, and n,m, k,l E 'Z. Translation 
invariance of the Euclidean length allows us to assume that the line segment under 
consideration, constituting a portion of the boundary of the domain of localization H, 
starts at the origin ( 0 , 0 ). 



Figure 1. Displays the components of geometric atoms. 








16 


Asymptotic Boundary Forms 


We observe that R{an, bm) = {dan + e6m, bmd ~^), 

length^ (i?San,bm) = {dan + ebmY + {bnid~^Y, 
the normalized vector v orthogonal to Rlan,bm equals 

V = (y{dan + ebmY + {bnid~^Y') {—bmd~^, dan + ebm), 

therefore we obtain 

length 5 ^^dist {R{ah^bl^^ 

= (^{{dak + ebl, bld~^), {—bnid~^, dan + ebm))Y 

= {—{dak + ebl)bm + {dan + ebm)bl)‘^ d~‘^ 

= {dab{nl — km) + eb‘^{ml — ml)Y d~^ = a^b‘^{nl — km)'^. 

The above calculation verihes fl35|) . It also shows fl3T|) . since plugging R = I, i.e. 
taking d = 1, e = 0 produces the geometric atom on aZ x 6Z. 

Let us recall that Gabor multipliers with symbols of the form b = xn are called 
Gabor localization operators and that they are denoted . The proof of Theorem 
11.21 the principal result of the current paper, is very lengthy and it makes use of all 
the auxiliary facts collected in Section 4. In order to facilitate the reading, we list the 
steps of the proof hrst, and then we present their proofs. 

Theorem 1.2. Let cf) be a generating function of a tight Gabor frame parametrized by 
lattice A and satisfying condition <h, and Q a A domain contained in . Then 

lim = ^BF{(t),n). 

R-^oo R Aa 


Steps of Proof: 

Step 1. We make a transition from lattice A to Z^ , where the computation is easier 
to handle. We construct a matrix A G SL{2,'R) and scaling parameters a, 6 > 0, such 
that Aa = ab and A(A) = S'a,6(Z^), where Sa,b is the scaling matrix with numbers 
a, b on its diagonal and zeros elsewhere. We will work with the 7? domain S~l{A{^l)) 
instead of the A domain G and with the generating function fi{A)(j) of the tight Gabor 
frame {{L{^)4>)'y}'yeSab{'z'^) instead of the generating function cf of the tight Gabor 
frame {^aIasa- Symbol /i denotes the projective metaplectic representation dehned 
on SL{2, M). We substitute the Gabor multiplier G^p^n by the operator s-i(n(o)) > 

which is unitary equivalent to it, and dehned by the formula 

Lf,{A)<p,s-f{A{n))f = E ( 36 ) 

uez2nS-f(A(n)) 

Once the unitary equivalence is verihed it is clear that 

PF{G^,n) = 


(37) 
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We substitute the boundary form BF{(f), f2) defined in (01) by the boundary form 
SF{^{A)(j), S~l{A{Q))) dehned in terms of the domain S~l{A{Q))) and the im¬ 
age of the original generating function 0 of the tight Gabor frame {0A}AeA under 
fi{A), 

N 

SF{fi{A)(j),S-l{A{Q))) = ^length(/i) '^dist{u,Pi)\{fi{A)(j),{fi{A)(j))s^,(u))\‘^, (38) 

i=l I'GUi 

where k, i = 1,N are the line segments constituting the boundary of S~l{A{il )), Ui 
is the unit vector orthogonal to k directed outside (y4(f2)), Pj = {w G | tc • Uj = 
0}, Pj = {i/ G Z^ 1 1 / • Uj > 0}. We show that in view of condition $ the form (138|) is 
well dehned and that 

SF(fi(/l)^,S„-;(yl(Sl))) = (39) 


Step 2. We use tight Gabor frames versions of Toeplitz and Hankel operators in order 
to express the projection functional 

as the square of the Hilbert-Schmidt norm of the matrix 

Mr{ui, U 2 ) = Xsyi(A(7jn))4^l)((h(^)0)s,,,(^2), (h(^)0)s,,,K))Xsyj(A(m))(^2), (40) 

where pi, z /2 £ Zi^. As the result we obtain the equality 


PF L 


^t,(A)4>,S-l{A{Rn)) 


R 


X5yi(A(_RO))'=(^l)-^(^l ^ 2 )XsyJ(A(RO))(^ 2 ), (41) 

1^2 


where 

P(z/) = |(/x(A)0, (/i(A) 0 ) 5 ^^^(^))p. (42) 

Next, we change variables and let them range over the dilated lattice ;^Z^. The right 
hand side of 1 1411) becomes 

^ ^5-i(A(o)y(G)^(^(i^i-i^2))Xs-i(A(o))(^2). (43) 

1 ^ 1 , t^2e-^z2 


Step 3. We split the boundary of S^l{A{il)) into its component lattice cycles, then 
into their individual line segments, and then we reduce the computation of the limit 
to the sum of the limits over the line segments constituting the boundary. We may 
assume that each boundary segment connects lattice points and does not have lattice 
points in its interior. If necessary we divide it into subsegments. Each sufficiently small 
neighborhood of a boundary segment gets represented as an unbounded vertical strip 
domain, for which expression 1 1431 ) becomes 

^ X[ 0 ,n) Xio,n) Y. F{xi-X2,yi-y2), (44) 
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and function H representing a given segment of the boundary is a linear function 
with rational slope of the form H{x) = ^x, with n > m, m > 0, n > 0, and m,n 
relatively prime. The graph of H constitutes the top portion of the boundary of the 
unbounded vertical strip domain {(x, t/) GM^|0<x<n, ?/< H{x)}. Points 
satisfying condition ^ > hf (^) lie above the graph of H, and points (^, ^) for 
which ^ < H are located below or on the graph of H. Function F is obtained 
out of via a lattice transformation of Lemma ITSl that brings 

the selected fragment of the boundary to the form described above. 

Step 4. We adjust the form of variables Xi,X 2 to the arithmetic form of the slope of 
H. We represent Xi,X 2 as Xi = nki +ri, X 2 = nk 2 + r 2 , where ki, ^2 are integers and 
^ 1 , ^2 G {0,1, 2,..., n — 1}. With new variables ki, vi, k 2 , r 2 expression 1 1441) becomes 


1 

R 


n—1 

ri, r2=0 fci, fe2€Z 


/ nki + ri\ 

ly R ) 


/ nk2 + r2 \ 

V R ) 


F{n{ki - k 2 ) + ri 

{yi I + {y 2 I y 2 <mk 2 +'^} 


rj.t/i -1/2)- 


(45) 


Step 5. Invariance of expression l l45l i with respect to variables yi,y 2 allows us to 
substitute the double summation with a single summation. We count the number of 
repetitions in the representation y = yi — y 2 and we place an appropriate factor that 
compensates them. Expression 1 1451 1 becomes 


1 

R 


n—1 

y~! y~! 

r\,r2=0 k\, fc2SZ 


f nki + ^ 

V R J 


X[0,n) 


/ nk2 + r2 \ 

V R J 


y^(i + l)F (^n{ki - k2) + n 

i>0 


r2,m(ki - k2) + 


'm 

—ri + 


L n 


1 


(46) 


'm ■ 
—F2 
L n J 



where square brackets denote the integer part of a rational number. 

Step 6. Form 1 1461 1 is convenient for making the passage to the limit. It occurs that 
taking the limit in 1 1461) with respect to R is just the same as performing summation 
with respect to k. We obtain 


n—1 


ri,r2=0 fcsZ i>0 


nk + Ti — r 2 , mk + 


m 

—ri + 1 

n 




(47) 


Step 7. We write expression ( 1471 ) in the form involving function Rt{s) 


n—1 n—1 

y] y y y (* + 1)F {kn + t,km + Rt{s) + i), 

t=o s=o fcez *>0 


(48) 
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where 

Rt{s) 


(t + s) + 1] - 5 ] for s e {0, 1} 

m + — (n — t)) + 1 ] — for s G {n — t,n — t + 1, ...,n — 1} 

(49) 


Step 8. We interpret expression ^481) geometrically. We make use of the structural 
features of lattice lines and we show that in fact 1 14811 equals 

\/n? + m? ^ dist {{x,y),G) F{x,y), (50) 

{x,y)eU 

where G = {{x, H{x)) | a; G M} is the graph of if, is the part of lying above G 

and dist is the Euclidean distance inside the plane containing lattice . Constant 
+ m? represents the contribution of vertical strip domain {(x, 1 /) G | 0 < x < 
n, y < H{x)} to the total length of the boundary of S~l{A{il)). 

Step 9. We put together the boundary forms (150|) of vertical strip domains corre¬ 
sponding to all line segments of the boundary of ^“^(^(n)) and we obtain 


lim 

R-^00 


PF L 




R 


SF{y{A)c^,S-i{Am). 


In view of fl37)) . and ([39]), formula (l5T]) concludes the proof. 


(51) 


Proofs of Steps: 

Proof of Step 1. We construct matrix A G SL{2,'R), satisfying property v4(A) = 
aZ X bli, a,b > 0, ab = Aa, by assigning values a(0,1), 6(1, 0) to the generators of A 
and then extending the assignment by linearity. The inverse of the scaling matrix Sa^ 
provides a transition from aZ x 6Z to Z^, = 0 ,^ x 6Z. Matrices A and Sa,b 

are the arithmetic tools needed for the transition from A to Z^. The unitary operator 
y{A), where y is the projective metaplectic representation dehned on S'L(2,M), is 
the analytic tool responsible for the transition needed on the level of the generating 
functions of tight Gabor frames. We observe that in view of Lemma 14.11 {ii) and 
definition ([36]) of T^(A)^, 5 y](A(o)) 

y{A)Gfp^Qli{A)* = G = Ry.{A)4>,s-l{A{n))- 

We conclude that (I57|) holds. The transition from A to Z^ is justihed as far the 
operator properties, i.e. the values of the projection functional, are concerned. 

The justihcation of the transition from A to Z^ as far as the geometric properties, 
i.e. the boundary form, are concerned, follows from Theorem 11.11 (ii). We obtain 


BF{(f>,n) = BF{y{A)4>,A{n)), 

but we still need to switch to SF{y{A)(l), S~l{A{^l))) dehned in fl55]l . In order to do 
that, we observe that the boundary form scales with respect to the action of Sa^b via 
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the area factor ab, and this allows us to hnish the proof of (j3^ 

Proof of Step 2. Since 

-^M(A)</>,5-j(A(RO)) = Gii{A)4>,A{RQ), 

Lemma [4.21 implies that 

where Mr{i'i^i' 2 ) is dehned in fHOjl . 

Proof of Step 3. This is the most tedious step of the proof. For the sake of notational 
convenience we will use symbol kl for the current domain under consideration. It will 
be clear from the context what it is at a given stage of the proof. Initially kl denotes 
. Our hrst target is to cut the kernel 

Kr{ui,U2) = ^xoc(z/i)F(i?(z/i - iy2))xn{i^2) 

into pieces with the help of a partition of unity of obtained out of quadrilaterals 
formed around the boundary segments constituting dfl and two open sets isolated from 
the boundary and representing the exterior and the interior of k2. 



Figure 2. Explains the process of traversing the boundary dQ. 

We traverse each component of the boundary of kl according to the orientation, keeping 
the interior on the right and the exterior on the left, and at each initial lattice point of 
the boundary segment li we place a sufficiently small rational line segment Sj with its 
middle being the initial lattice point. We chose Si in such a way that it is transversal to 
both li and the boundary segment li preceding it, and that both exterior and interior 
angles with li and k are smaller than vr. We assume that segments Si,i = 
are so small so that they intersect the boundary dQ exactly at one point. We assume 
that they are positioned in such a way that one of the endpoints of s* is inside the 
interior of kl and the other in the exterior of k2. Let Ir be the segment following li. We 
form quadrilaterals Wi out of consecutive segments s*, Sr attached at the beginning 
and at the end of li, and the segments joining their endpoints, both endpoints inside 
the interior of kl or both in the exterior of k2. We also assume that the sizes of the 
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transversal segments are so small so that the segments joining the endpoints of Si and 
Sr are contained either in the exterior or in the interior of . We include Sj inside Wi , 
but not Sr- Adding Sr to Wi we would obtain a closed quadrilateral, but we need to 
form a family of pairwise disjoint sets, and Sr is already included in Wr ■ 



Figure 3. Illustrates the construction of quadrilaterals Wi- 


We dehne a partition of out of quadrilaterals Wi^i = 1,..., forming a neighbor¬ 
hood of the boundary dVt and two open sets Ue = Uili Uj = Vt \ IJ^i bhj 
representing the exterior and the interior of Let m* = = XUe > 

mj = Xu I ■ We dehne 


Kr{^1,U2) 

K]^{ui,U2) 


N 

mfc(z/i)A'ij(z/i, V 2 )mi{v 2 ), 

k,l=l 


N 

mij (1/1) iL/j (z/i, z/2) m/(z/2), 

i=i 


N 

'^mk{ui)Kji{ui,U2)mi{u2), 

k=l 


mE{ui)KR{ui, U2)mi{u2). 


We observe that 


IV 


kr = Y, Ki 

j=i 


(52) 


We do not need terms with factors m/(i/i) and mE{y 2 ) in the above decomposition, 
since f// fl = 0 and fl = 0. We know that dist([//, > 0, dist([/E, fl) > 0, 

therefore Lemma 14.61 allows us to conclude 


lim 

R-^oo 




(53) 


!21,I^2G4A 


for J = . Since dist(II4, lid) > 0 for /c 7 ^ / such that IkJi are not neigh¬ 

boring line segments, again by Lemma 14.61 we obtain 


lim 

R -^00 


mk{ui)KR{ui,U2)mi{u2) 


0 . 


(54) 
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If A; 7 ^ /, but lk,li are neighboring line segments, then Lemma Wl\ applies and we 
conclude again 


lim 

R-^oo 


mk{iyi)Kii{ui,U2)mi{u2) 


0 . 


(55) 



Figure 4. Explains the usage of Lemma [4.71 for neighboring segments lk,h- 


Formulae allow us to conclude that only terms of the form 




contribute to the limit. 

Let us recall that we assumed that each boundary segment Ik connects lattice points 
and does not have lattice points in its interior. We may apply Lemma 14.51 in order to 
bring each segment Ik to the form {x,H{x)), with 0 < a: < n, H{x) = ^x, n > 0, 
m > 0, n > m, n,m relatively prime, with the image of placed below the graph 
of H, and the image of placed above the graph of H. Under the transformation 
of Lemma 14.51 quadrilateral Wk becomes Wk ■ With the help of Lemma 14.71 we bring 
Wk to a form of a bounded vertical strip domain with the top portion of the boundary 
represented by a line tk and the bottom portion by a line bk . Lemma 14.61 allows us to 
substitute the bounded vertical strip domain we have just obtained by a rectangle with 
the top boundary represented by a horizontal line y = r, with r G Z, placed above tk, 
and the bottom boundary represented by a horizontal line y = s, with s G Z, placed 
below bk- 

















H.G. Feichtinger, K. Nowak, M. Pap 


23 



Figure 5. Illustrates the usage of quadrilateral II 4 . 
In the last step we switch to the unbounded strip domain 

= {(x, y)\Q < X < n,y < H{x)}, 


with its complement 

= {(x, y) \ 0 < X < n,y > H{x)}. 


We have the splitting 

- «^2))xo(P2) = 

X{(a:,j/) I 0<a;<n,j/<r}(Pl)XO'=(Pl)-^(-R(i^l ^2))XO(P2)X{(x,y) | 0<x<n,y>s}(^2) (56) 

"hXKajjj/) I 0<a:<n,j/>r}(^l)Xf2'=(^l)-^(-R(^^l ^2))Xo(^2)X{(®,y) | 0<x<n,y>s}(^2) (57) 

"hXKajjj/) I 0<a:<n,j/<r}(^l)XO'=(Pl)7^(7?(l2l P2))XO(P2)X{(3;,j;) | 0<x<n,y<s}(^2) (58) 

I 0<a;<n,j/>r}(Pl)XO'=(Pl)7^(7?(l2l ^2))Xfii^2)X{{x,y) \0<x<n,y<s}{^2) (59) 

We need to show that the normalized sums coming out of terms fl57)) . fl58l) . fl5^ have 
zero limits. Lemma 1^^ applies to both fl57)l . fl58|l . but fl5^ needs to be treated sepa¬ 
rately. After the change of variables the normalized sum of (l59ll becomes 

^ F{xi-X2,yi-y2)- 

0<xi,X2<Rn yi>rR 
y 2 <sR 
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In the proof that its limit is zero as i? —)■ cxo we may assume that R is an integer. Let 
u = min{r, —s} and let * denote the convolution on Z. We have 

^ F{xi-x 2 ,yi-y 2 ) XI “ ^ 2 ,2/i + 1 / 2 ) 

0<x\,X2<Rn yi>rR 0<xi,X2<Rn yi,y 2 >uR 

y 2 <sR 

^ F{xi-X2,y)iy-2uR + l) 

0<xi,X2<Rn y>2uR 


<n X (Fi;y)iy + i)*^2^§^) 

y^R\ [RnyR {RnyR / 

'^F{x,yyy + l) 0 , 

^ ^ R-^oo 

y>2uR xG.1^ 


because 2uR —)■ 00 as i? —)■ cxd , since u > 0, and 


EE F{x,y){\y\ + 1 ) < cx). 


Proof of Step 4. With the new representation of Xi,X 2 the summation done in flUD 
becomes fl^5|) . It is enough to perform substitutions and then simplify the resulting 
expressions. 


Proof of Step 5. We have 


mri 

yi > mki H-, i.e. yi > mki + 

n 

mr2 

y 2 < mk 2 H-, i.e. y 2 < mk 2 + 


n 


mri 

n 

mr2 
n . 


+ 1 


therefore 


1/1 - 2/2 = rn{ki - ^ 2 ) + 


mri 


L n 


+ 1 


mr2 


L n 


+ h 


for some i = 0,1, 2,.... A given value i occurs for exactly i + 1 pairs 1 / 1 , 1/2 • 

Proof of Step 6 . We move summations over ki,k 2 in fHbll inside and summations 
over ri,r 2 -,i outside. Our target is to identify convolution kernels dehned in terms of 
variables fci,/c 2 and then, keeping variables ri,r 2 ,i hxed, take the limit with respect 
to R. For a rational number g by \q] we denote the smallest integer larger or equal 
to q. We know that 

nkj + Tj 

0 < ' <n,i = 1,2. 

R 

We observe that ki ranges over the integer interval 


0, \R - 

n 

We dehne convolution kernels Kr^^r 2 ,i as 


a:. 


,1 = 1 , 2 . 


(k) = F ink + Ti — r 2 , mk + 


m 


n 


-ri + 1 


m 


n 


-r2 


+ i 
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Lemma 14.41 guaranties that as i? —?• cxo the expression 

\{l — li] “ ^2)X[o,rR-^]-l] (^l) X[o,rR-^]-l] (^ 2 ) 

^ fci,A:2GZ 

tends to ^ri,r 2 A^) ■ proved that fl47D is the limit of fl4^ . 

Proof of Step 7. Each number kn + t, t G — 1} has two possible rep¬ 

resentations. The hrst one as fcn -|- ri — r2 with ri — r2 = t and the second one 
as {k + l)n -|- ri — r2 with ri — r 2 = t — n. The hrst case occurs for ri = f -|- s, 
r 2 = s, s E {0,1, ...n — f — 1} and the second case for ri = s — {n — t), r 2 = s, 
s G {n — t,n — t -\-1, ...,n — 1}. For f = 0 we have only one representation of kn + t as 
fcn + ri—r2 with ri = r2 = s, s G {0,1, ...n —1}. Function Rt{s) incorporates all these 
relations and allows us to subsitue summations with respect to ri,r 2 by summations 
with t,s, therefore fITTD becomes (1451) . 

Proof of Step 8. We hx summation variables fc, t of expression (1451) and we consider 
the effect of the summation done with respect to s,i. By Lemma [4.31 we know that 
function Rt{s) takes two values \j^t -f l] and \j^t -|- l] -|- 1, therefore we conclude that 
points {kn + t, km + Rt{s) + i), s = 0,1,..., n — 1, i = 0,1,2,, represent a discrete 
vertical half line consisting of lattice points of A starting directly above the graph 
G. Lemma [4.31 also tells us that value + l] is taken n [[^t -|- l] — times and 
value -|- l] -|- 1 the remaining n — n -|- l] — times. The hrst point of the 
discrete half line, located right above the graph G, corresponds to parameter values 
i = 0, Rt{s) = + l], and it is repeated inside formula (1451) n {[^t + l] — 

times. The jth point of it, j >2, with counting done upwards, corresponds to values 
i = j — 1, Rt{s) = [^t + l] and i = j — 2, Rt{s) = -|- l] -|- 1, and in (14511 it comes 

with multiplicity — 1 + [^t -|- l] — . Let dj,Vj represent the Euclidean and 

the vertical distances from the jth point, j > 1 of the discrete half line to the graph 
G. Similarity relation of the triangle representing distances di,Vi with the triangle 
with vertices (0, 0), {n, 0), (n, m) gives 

dj n 

Vj \/nA+AR 

Since the multiplicity inside (145|) of the jth point of the discrete half line equals nvj, 
and variables k, t parameterize all discrete vertical half lines of U we conclude that 
(1451) may be expressed as (150|) . 

Proof of Step 9. It is enough to combine together the outcomes of steps 1-8, i.e. all 
of the intermediate stages of the reduction process. 

The proof of Corollary 11.31 follows directly from Theorems 11.1111.21 

Corollary 1.3. For any lattice A C satisfying condition Aa < 1, any generat¬ 
ing function (j) of a tight Gabor frame {^aIasA; o.ny A lattice domain 12, and any 
a,h > 0 satisfying ab = Aa, there are a generating function (j) of a tight Gabor 
frame {0A}AGaZxra and a aZ x bZ lattice domain satisfying ||0 ||i,2(r) = ||0||a2(k), 
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Area{Q) = Area{fl), PF{G^^rq) = PF{G^j^^), for all R> 0, and also BF{(j),Q) = 
BF{^, fi). The rates of convergence of to ■^BF{(j), f2) and to 

■PpP^f), fi) are the same. 

Proof. It is enough to take A G S'L(2,M) transferring A onto aZ x 6Z, 0 = fr{A)(l), 
n = A(f2), and apply Theorem ll.li The existence and the form of the limits follows 
from Theorem 11.21 


4. Auxiliary Facts and their Proofs 

Transformation properties with respect to the metaplectic representation. 

A comprehensive presentation of the metaplectic representation from the point of view 
of phase space analysis is contained in Folland’s book [19]. The book by Lang [30] is 
an extensive reference on S'L(2,R). The Heisenberg group is the group obtained 
by defining on the product 

(z, t) ■ {w,s) = + w,t + s - , 

where x, w G f, s G M and u is the symplectic form defined on i.e. 

uj{z,w) = z^Jw, J = 

The Schrodinger representation of the group acting on is then dehned by 

t)f{y) = - x). 

We write z = {x,^) when we separate the position component x from the momentum 
component ^ of a point z of the phase space The group S'L(2,M), consisting of 
2x2 matrices with real entries and determinant 1, acts on via automorphisms 
that leave the center {(0,t) | ^ ^ 1^} of pointwise fixed, 

A ■ {z, t) = {Az, t). 

For any fixed A G S'L(2,M) there is a unitary representation of acting on 
defined as the composition 

pA{z,t) = p{A- {z,t)), 

with its restriction to the center of being a multiple of the identity. By the Stone- 
von Neumann theorem pA is unitary equivalent to p, i.e. there is an intertwining 
unitary operator p{A) acting on L^(M) such that for all {z,t) G 

Pa{z, t) = p{A) o p{z, t) o /i(A)"h 

By Schur’s lemma, p is determined up to a phase factor e®®,s G M. It turns out that 
the phase ambiguity is really a sign, so that p lifts to a representation of the double 
cover of the group S'L(2,M). The constructed representation of the double cover of 
SL{2,M.) is called the metaplectic representation. 


0 1 

-1 0 
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The representations p and p can be combined and give rise to the extended meta- 
plectic representation /ig, the composition of p, dehned on with p, dehned on 
the double cover of S'L(2,M). From the point of view of the interpretation as a phase 
space action, the phase factors do not matter, therefore we remove them and treat p 
as a projective representation of SL{2, M), and Pe as a projective representation of the 
semidirect product xi S'L(2,M) with the group law 


{z, A) ■ {w, B) = {z + Aw, AB). 


(60) 


The extended metaplectic representation provides all affine transformations of the 
phase space . For (z, A) G x SL{2, M) the unitary operator pe (z, A ), dehned up 
to a phase factor, expresses the analytic action on L^(R). The affine geometric action 
on is expressed by the law 


(z, A)w = Aw + z, where w G 


(61) 


The extended metaplectic representation is a convenient setup for performing compu¬ 
tations involving compositions of the Schrodinger and the metaplectic representations. 

Both Gabor multipliers (Ej) and lattice boundary forms (jll) have natural transfor¬ 
mation properties with respect to the projective metataplectic representation. These 
properties are important ingredient of our proofs. We will deduce them out of the the 
fundamental lemma formulated below. 

Lemma 4.1. Let A C be a lattice, {0a}aga o tight Gabor frame with the generating 
function 0, A G SL{2,M.), and b G 1°°{A). Then 

(i) F = AA C is a lattice, {(/i(A)0).y}.^gr « tight Gabor frame with the generating 
function p{A)(j), and 


(0, (fx) = (/i(A)0)^) for 7 = AA, (62) 


(a) bo A ^ , and 


pi^A'jG (phpi^A') 1 • 


(63) 


Proof. Clearly F = AA is a lattice, since A is linear and invertible. Group law 
fl60|) and affine action rule (16T]) allow us to identify the phase space with the 
subgroup {{{q,p), /) | G M} of x SL{2, M). Since (0, A)(A, /)(0, A~^) = (AA, I ), 
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by substituting 7 = AX we obtain 

^{A)GtMArf = ^MA)(/,fi(A(<i'A)>M(-4)(.#.A) 

AeA 

AeA 

= Y. M=((0, ^)(A, /)(0, A-^))MA)<l‘)tte({0, A)(\, /)(0, A-^))MA)4‘ 

AeA 

yer 

yer 

^ lj.{A)ifi,boA~^ f ■ 

The above calculation shows that {(/i( 74 ) 0 )^}^gr is a tight Gabor frame, it is enough 
to take the constant function equal to 1 for b. It also verifies formula fl 6 ^ . Formula 
fl 6 ^ follows by a similar calculation, 

(/i(kl)0, (/i(kl)0)^) = (0,/ie((O,kl“^)(7, J)(O,kl))0) = (0,/ie(kl“^7, J)0) = (0,0 a). 

Symbolic calculus of Gabor multipliers. Let us assume that A C is a lattice, 
and {0A}AeA a tight Gabor frame defined on it. Let us define the mapping IF^ : 
L^(M) —)■ /^(A) by the formula 

W^f{X) = {f,cl>,). 

Tight frame properties of {^yjAeA imply that fF/, is an isometry, and that the operator 
: /^(A) —)■ P{A) defined as 

P^h(A) = 

tiGA 

is the orthogonal projection onto the range of lF,ji. Gabor multiplier is parametrized 
by the generating function 0 of a tight Gabor frame {0 a}aga and a symbol b G 1°°{A). 
Let us recall that it is a bounded operator acting on L^(M) defined as 

aga 

It is convenient to describe Gabor multiplier in terms of the Toeplitz operator 
G(f>,b = PipMiiPif, acting on /^(A), Mi, denotes the operator of multiplication by b. 
The isometry 1F<^ allows us to identify with the upper left corner of the ma¬ 
trix representation of the operator with respect to the orthogonal decomposition 
1^{A) = W^{L‘^{R)) © lF</,(L^(M))-‘-. Basic properties of the symbolic calculus of Gabor 
multipliers can be deduced out of the properties of the symbolic calculus of Toeplitz 
operators. Hankel operator = {I — P<j>)MbP(t, acting on /^(A) measures to what 
degree the mapping b -A- Q,f,^b fails to be a homomorphism. The algebraic formula 

G(j>,bib2 Gij>,b\_G(f>,b2 
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expresses this relationship quantitatively and it is the main conceptual ingredient of 
the argument that allows us to write down the projection functional 

PF{G^,n) = PF{g^,n) = tr {g^,n (/ - g^,n)) 
as the square of the Hilbert-Schmidt norm the matrix 

where Ui, z /2 G A. 

Lemma 4.2. Let {^aIaga be a tight Gabor frame and C A a finite set. Then 
PF{G^^q)= Xn-{^i)\{(pu2,(pui)\^Xn{^2)- 


Proof. The non-zero eigenvalues of the localization operator G^^q coincide with the 
non-zero eigenvalues of the Toeplitz operator g^^Q = P^pM^^Pij, and the non-zero eigen¬ 
values of the operator . The hrst fact follows from the identihcation 

of with the upper left corner of the matrix representation of g^^Q with respect to 
the decomposition of F{A) into the range of and its orthogonal complement. The 
second fact follows since for a compact operator T the non-zero eigenvalues of TT* 
and T*T are the same. We observe that 

P A"(G<j!),o) = PF{g<f,,n) = tr {I — g 4 >,n)) = tr (/ — 

and that 

^xn^4>^xni^ ~ ^xn^4>^xn) ~ ~ ^xn)^<P^xn ~ ^xn^4>^xn<^^<t^xn- 

Therefore we obtain 

PF{G^,n) = Y 0^i)rAo(P2), 

and this hnishes the proof. 

Lattice slopes of rational lines. For an integer t we dehne the t-slope of the line 
y = ax b at an integer argument s as 

St{s) = U{a{t -|- s) -|- 6) — l{as -f 6), (64) 

where U{x) is the smallest integer larger than x, and l{x) is the largest integer smaller 
or equal to x. The t-slope at s is simply the smallest difference between integer values 
above the graph at t s and below or on the graph at s. We interpret it as the 
lattice slope corresponding to making t-steps to the right of s. We do not normalize 
the t-slope, i.e. we do not divide it by the number of steps. As we have already seen 
function Rt dehned in fl49l) is the principal analytic component of the boundary form. 
It occurs that it may be interpreted as the t-slope of the rational line y = "^x. Indeed, 
let St{s) be the f-slope of the line y = ^x at s. Direct inspection shows, that values 
Rt{s) and St{s) coincide for s G {0,1, 2,..., n —1}. Function St is periodic with period 
n and we may regard it as dehned on the cyclic group . 
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Our primary geometric concern are the values of lattice slopes and the frequencies 
with which they occur. Observe that Ro{s) = 1 for all s G {0,1, 2, n — 1}. If n = 1, 
then the only possible value of f is 0 and again 1 is the only value of Rt. If however 
n>2 and t 0, then Rt takes precisely two values. The next lemma describes those 
values and the frequencies with which they occur. 

Lemma 4.3. Let m,n be a pair of relatively prime integers. Assume that n > 2. Let 
t G {0,1, 2,..., n — 1} be a fixed number. Function Rt{s) defined for arguments s G 
{0,1, 2 ,..., n — 1} by formula ([7^ takes two distinct values + l] o^nd + 

Value \j^t + l] is taken n ([^t + l] — ^t) times, and value [^t + l] + 1 is taken the 
remaining n — n + l] — times. 

Proof. We know that Rt and St are equal. It is therefore enough to prove Lemma 
with St instead of Rt. Let s G {0,1,2, ...,n — 1}. Line y = '^x crosses vertical lines 
X = s at points of the form (s, I + ^), where I is an integer and r G {0,1, 2,..., n — 1}. 
Numbers m,n are relatively prime, therefore each value of r occurs precisely once for 
an appropriate value of s G {0,1,2, ...,n — 1}. Observe that the set of values Sfis), 
s G {0,1, 2,..., n — 1} is the same as the set of f-slopes at 0 of the lines y = '^x + ^, 
r G {0,l,2,...,n — 1}. Indeed an integer shift of coordinates allows us to view each 
segment starting at (s, ^s) and ending at (t + s, + s)) as a segment starting at 
(O, and ending at (t, ^t + for an appropriate value of r. Line y = ^x crosses 
the vertical line x = t at a point I + ^ with I an integer and mG{ 1 , 2 , ...,? 7 , — 1 }. All 
lines y = ^x + ^ with r G {0, 1,2, ...,n — u — 1} have the same f-slope at 0 as the line 
y = ^x. The f-slope jumps up by 1 for r = n — u and keeps this value for all all the 
remaining r G {n — u, n — u + 1, ...,n — 1}. We conclude that function St takes two 
values + l] and [^t + l] +1. The hrst value is taken n — u times and the second 
value u times. We need to verify that n — u = n(^ + l] — ^tj. Indeed, ^ ^ 

therefore mt = nl + u, [^t + l] = Z + 1 and n {\fi^t + l] — = n{l + l)—mt = n — u. 

Approximation to the identity by Fejer’s kernel. In the lemma that follows we 
quote a well known approximation to the identity property of the Fejer’s kernel. We 
translate the original property from the group of one dimensional torus to the group 
of integers. For f,g& fiifL) \yy f * g we denote the convolution of / and g, dehned as 

f * 9 {k) = ^f{k-l)g{l). 

Lemma 4.4. |27] If K E then 

(^*X[o,7 v-i],X[o.v-i]> = ^K{k), 

fcgZ 

where {■, ■) stands for the inner product of . 

Invariance properties of the restricted kernel . Let us recall that kernel 
Kr{vi,V 2) = ^Xnfiv)F{R{ni - z/2))xo(i^2), 
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where is a domain, z/i,z /2 £ and F{i/) = |(/r( 74 ) 0 , (/i(y 4 ) 0 ) 5 _^ was 

defined in dH]). This section presents invariance properties of kernel needed at 
various stages of the process of reduction. Let Aut(Z^) = Aut(-^Z^) = {—1, +1}^ x S 2 , 
be the group of automorphisms of I? (or ;^Z^), i.e. the semi-direct product of sign 
changes and permutations of variables (see [32] page 110). Let ^1? x Aut(Z^) be the 
group of affine transformations of ^1? consisting of translations and automorphisms of 
-^Z^. The elements of ^1? x Aut(Z^) are represented as pairs (/i, h), where /i G 
h G Aut(Z^). The group law has the form (/xi, hi)(/i 2 , ^ 2 ) = (hi(p 2 ) + hi) ^ 1 ^ 2 )- 

Lemma 4.5. Let V,W C and let r = (/i, h) G x A?/f(Z^). After the change 
of variables pi = r“^(z/i), ^2 = f~^(p 2 ) the restricted kernel Kr 

^XF(hi)Xo=(hi)^(^(hi - h 2 ))xo(h 2 )xiv(h 2 ) 

becomes 

-^Xr{V){j^l)Xr{n)4’^l)F{h~^{R{jyi - P 2 )))Xr( 0 )(z^ 2 )Xr(W)(i" 2 ). 

For any 1? domain Vl and any line segment I C dQ it is possible to choose a trans¬ 
formation T G X Aitf(Z^) such that t{1) is a segment of the graph of H{x) = , 

where n>m, m> 0, n > 0, m,n are relatively prime, and the portion of T(f 2 ) close 
to t{1) is placed below t{1) , the portion of r^fiy close to t{1) is placed above t{1) . If 
the line segment I does not contain lattice points in its interior, then we may assume 
that t{1) = {(x, H{x)) I 0 < a: < n} . 

Proof. The proof of the first part, the formula for the coordinate change is a straight¬ 
forward computation which makes use of the fact that = {—h~^{fi),h~^). The 
proof of the second part follows the process of inspecting the list of all possible posi¬ 
tions of the segment I, which takes into account the placement of fl and relative 
to /, and indicating in each case the coordinate change r needed to accomplish the 
target. It is also a direct computation. 

Asymptotic limits of the restricted kernel Kji. The following two lemmas are the 
main technical tools behind the reduction process of general lattice domains to vertical 
strip domains. In the first lemma we deal with separated supports of variables i^i, 1 ^ 2 - 
The second lemma is more delicate, it treats the case of variables i'i,i '2 restricted to 
bounded cones located outside and inside , with their sides being segments of rational 
lines, and their common vertex being a lattice point of the boundary of 

Lemma 4.6. Suppose that there is 5 > 0 such that for all sufficiently large R non¬ 
negative kernel Gji{h'i,h' 2 ), ^ 1,^2 £ ;^Z,^ satisfies 

Gr{i 2 i, z/ 2 ) < Kr(ui, U 2 ), and Gr{ui, z/ 2 ) = 0 for |z/i - Z/ 2 I < 6. 
lim Gr{ui,i 22 ) = 0. 

R-^00 ^^ 


Then 
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Proof. For each z/2 E fl we have 


^ Gr{ui,U 2)< Xn4^i)F{R{i^i-i^2))xn{i^2) 

yi€^7? 

ki— i^2|><5 Wi—i'2\>S 


< 


Xn{i^2) 

R 


E 

I2GAz2 

IH><5 


Xo(^2) 

i? 


E s 

agz^ 

|A|>(Si? 


Xo(^22 ) 

(5/?2 


E 

AgZ2 

\X\>SR 


Let Qr= [^, ^] X [^, ^]- We observe that 


^ Xni.^2) 

V R‘^ 

^2GjjZ2 


z/2 + Qr 

< 


i'2€-^z^rn 




where for r > 0 by we denote {m G | dist(M,n) < r for some n G f2}. Since 
^Aez2 -^(A)|A| < 00 we obtain 


E 

1^1,1^26 4 ^ 


G_r(i^i, z/2) < - 


f2 V2fl 


E 

Aez2 

|A|><5_R 


^ 0 . 

R —^00 


Lemma 4.7. Suppose that there are two closed, bounded cones Ci,C 2 , Ci C 
C2 C f2, with their sides being segments of rational lines, their apertures smaller than 
71 and their intersection being their common vertex, a lattice point of the boundary of 
f2. Suppose also that for all sufficiently large R nonnegative kernel Gr^ui, U 2 ), z/i, z/2 G 
-LZ^ satisfies 

Gr{i 2 i, z/2) < Kr{ui, z/2), and Gr(z/i, z/2) = 0 for (z/i, 1 x 2 ) ^ Gi x C'2. 


Then 


lim 

R^oo 




Gr{Ui,U2) 




0 . 


Proof. We observe that in view of Lemma 14.61 we may substitute cones C'i,C'2 by 
restricted cones G( = GiDDsiv), = G 2 r]Ds{v), where Ds{v) is the Euclidean disk 

with radius 6 and center v, the common vertex of C'i,C2. We may take the radius 6 
arbitrarily small. We choose a lattice line I passing through v and separating Gi and 
C2. Lemma 14.51 allows us to assume that line I has the form H{x) = ^x, m > 0, 
n > 0, m < n, gcd(m,n) = 1, that the origin is the common vertex of Gi,G 2 , and 
that Gi lies above I and C'2 below /. 

If m 7^ 0, i.e. line I is not horizontal, then we extend the lattice I? to lattice A, 

A= (—Zx-Z) U (—,— ) + (—Zx-Z) , (65) 

\m n J \2m 2nJ \m n J 

and kernel F to kernel F, 


F(A) = F(A), 


(66) 
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where A G Q = x A), A G A + Q. We observe that line I becomes a 

horizontal line with respect to A. We obtain 

i ^ ^ F(.R{P, - h)) = j Y1 bA.-Aj) 

i^isCfn-iz^ AisCf'^nA 

j/ 26 C|n^z 2 i>2GC|niA A2eCf'5nA 

E bA. + Aj) (67) 

Ai,A2G(CiU-C2)“nA 

In the next step we bring the snm fl67|) to a computable form. We introduce an integer 
cone 0 < ?/ < R6, —My < x < My, x, y G Z with a sufficiently large aperture 
M G Z in order to capture all points of (Ci U — 6 * 2 )'^'^ fl A of fl^ . Constant a is the 
magnihcation factor needed to switch from A to an integer lattice. Parameter R = Ra 
of the integer cone accounts for this magnihcation. We estimate (l67|) by 

^ Y1 F (^{xi+X2,yi+y2)] , (68) 

o<vi,v2<rs ^ ^ 

“ “ -My2<X2<My2 

where F is F adjusted to the new coordinate system adapted to representing I as a 
horizontal line. Let x = xi +X 2 , where —Myi < xi < Myi, —Mj /2 < X 2 < My 2 . 
Then —M(|/i + 1 / 2 ) < x < Miiji + 7 / 2 ) and x is represented as xi + X 2 for at most 
2Mmin{|/i, 1 / 2 } + 1 pairs xi,X 2 . Therefore we may estimate (168|1 by 

^ Y1 F (^{x,yi + y2)] {2Mmm{yi,y2}+ 1). (69) 

0<yi,y2<RS-Miyi+y2)<x<M(yi+y2) ^ ' 

Let y = yi + y 2 ) where 0 < |/i ,|/2 < R5. Then 0 < ?/ < 2R5 and y is represented as 
Hi + 2/2 for exactly y + 1 pairs 2 / 1 , 2 / 2 - We may estimate fl6^ by 

^ E E F(l{x,y)](2My + l){y+l). (70) 

^ 0<y<2R5 -My<x<My A / 

Since y + I < 3R6 for sufficiently large R, and 2My + 1 < 2M{y + 1) we conclude 
that flTU]) may be estimated by 

56Ma ^ F 2/)l iv + !)• 

This concludes the proof since constants M, a depend only on cones Ci,C 2 , condition 
<h guaranties that '^xyezF{j{x, 2/)) (2/ + 1) < oo, and we are allowed to take 6 
arbitrarily small. 
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